Image processing apparatus and image processing method

ABSTRACT

In an image processing performing local tone correction on an image, tone correction on an object (face) area can be suppressed from becoming nonuniform. The apparatus is provided with an acquisition unit ( 102 ) that acquires coordinate information on an object area, a division unit ( 103 ) that divides the inputted image into a plurality of divided areas, a first decision unit ( 105 ) that decides atone correction coefficient for each of the divided areas, a second decision unit ( 106 ) that decides a tone correction coefficient for the object area based on coordinate information on the object area and the tone correction coefficients for the plurality of divided areas, and an image processing unit ( 107 ) that performs tone correction processing on the object area by applying thereto the tone correction coefficient decided by the second decision unit in a uniform manner without depending on coordinates thereof.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional of application Ser. No. 13/968,528,filed Aug. 16, 2013, which is a divisional of application Ser. No.12/980,869, filed Dec. 29, 2010, now U.S. Pat. No. 8,532,426, the entiredisclosures of which are hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus and animage processing method of performing tone correction on a moving imageor a still image being inputted.

2. Description of the Related Art

An area in which the face of a person is displayed (hereinafter,referred to as a “face area”) in a moving image or a still image is animportant area to which viewers pay particular attention. Therefore,with a photographing device such as a digital camera, etc., thedetection of the face area is generally carried out.

On the other hand, with display equipment such as a television, etc.,local tone correction is generally carried out for the purpose ofenhancing contrast, reducing local extinction of tone in dark region andbright region, or the like. For example, there is cited processing suchas one in which tone correction is carried out by dividing an image intoa plurality of areas so as to obtain local statistical information(e.g., brightness information) of the image, and locally controlling agamma curve that is used for tone correction, based on the statisticalinformation thus obtained.

In the patent gazette of Japanese patent application laid-open No.2009-59119, there is disclosed atone correction system in which an imageis divided into a plurality of block areas in order to obtain a suitabletone for the face part of a person, and the brightness of an input imageis corrected based on the brightness information of the plurality ofblock areas and the brightness information of the face part.

SUMMARY OF THE INVENTION

However, in the above-mentioned tone correction system, the tonecorrection is carried out based on the brightness information which isdetected for each of the divided areas, and the brightness informationof the face part, so tone correction, being uniform over the face part,has not necessarily been made. As a result, the tone correction of theface area, to which viewers pay attention, might become nonuniform, thusgiving rise to a fear that an odd or uncomfortable feeling might begiven to the viewers. For example, in cases where there exists a blackobject in the vicinity of the left side of the face part, the left sideof the face might be influenced by the tone correction to the blackobject, so that tone correction might become nonuniform on the rightside and the left side of the face.

Accordingly, an object of the present invention is to suppress tonecorrection to an area in which a specific kind of object such as theface of a person, etc., exists, from becoming nonuniform, in an imageprocessing apparatus and an image processing method of performing localtone correction on an image.

The present invention in its first aspect provides an image processingapparatus which performs tone correction processing on an imageinputted, the image processing apparatus comprising:

an acquisition unit that acquires coordinate information on an objectarea which is an area where a predetermined kind of object exists in theinputted image;

a division unit that divides the inputted image into a plurality ofdivided areas;

a first decision unit that decides a tone correction coefficient foreach of the divided areas related to the tone correction processingbased on brightness information of each of the plurality of dividedareas;

a second decision unit that decides a tone correction coefficientrelated to tone correction processing on the object area based oncoordinate information on the object area and tone correctioncoefficients for at least divided areas which have a common portion withthe object area, among the plurality of divided areas; and

an image processing unit that performs tone correction processing on theobject area in the inputted image, by applying thereto the tonecorrection coefficient decided by the second decision unit in a uniformmanner without depending on coordinates thereof, and that performs tonecorrection processing, in which weighted interpolation is carried outaccording to coordinates by the use of the tone correction coefficientfor each of the divided areas decided by the first decision unit, onareas other than the object area in the inputted image.

The present invention in its second aspect provides an image processingmethod which performs tone correction processing on an image inputted,the image processing method comprising:

an acquisition step to acquire coordinate information on an object areawhich is an area where a predetermined kind of object exists in theinputted image;

a division step to divide the inputted image into a plurality of dividedareas;

a first decision step to decide a tone correction coefficient for eachof the divided areas related to the tone correction processing based onbrightness information of each of the plurality of divided areas;

a second decision step to decide a tone correction coefficient relatedto tone correction processing on the object area based on coordinateinformation on the object area and tone correction coefficients for atleast divided areas which have a common portion with the object area,among the plurality of divided areas; and

an image processing step to perform tone correction processing on theobject area in the inputted image by applying thereto the tonecorrection coefficient decided by the second decision step in a uniformmanner without depending on coordinates thereof, and to perform tonecorrection processing, in which weighted interpolation is carried outaccording to coordinates by the use of the tone correction coefficientfor each of the divided areas decided by the first decision step, onareas other than the object area in the inputted image.

According to the present invention, in an image processing apparatus andan image processing method of performing local tone correction on animage, it becomes possible to suppress tone correction on an area inwhich a specific kind of object such as the face of a person, etc.,exists from becoming nonuniform.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an image processing apparatus according toa first embodiment and a second embodiment.

FIG. 2 is a view for explaining image processing on an image including aface area according to the first embodiment.

FIG. 3 is a view showing gamma curves according to average luminance orbrightness values, used in the tone correction of the first embodiment.

FIG. 4 is a view showing a virtual gamma coefficient k′.

FIG. 5 is a flow chart of processing for deciding a tone correctioncoefficient in the first embodiment.

FIG. 6 is a flow chart of tone correction processing in the firstembodiment.

FIG. 7 is an example of a plurality of face areas overlapping each otherand an inclusion area which includes them in the first embodiment.

FIG. 8 is an example in which a plurality of face areas exist in thesecond embodiment.

FIG. 9 is a view explaining the calculation of an average value of gammacurves in the second embodiment.

FIG. 10 is a block diagram of an image processing apparatus according toa third embodiment.

DESCRIPTION OF THE EMBODIMENTS First Embodiment

FIG. 1 is a block diagram showing the construction of an imageprocessing apparatus according to a first embodiment of the presentinvention. This image processing apparatus is an apparatus that outputsimage data to which the result of detection of a face area by means of adevice and/or a program that detects an area (face area) in which animage of the face of a person in an image exists is added as metadata,while performing tone correction thereon.

This image processing apparatus can be applied to photographing devicessuch as digital cameras, digital camcorders, etc., video display devicessuch as digital televisions, monitors, etc., and video output devicessuch as recorders, players, game machines, etc. Here, note that thisembodiment is an example of a case in which “a predetermined kind ofobject” in the present invention is “the face of a person”, and “a facearea” in this embodiment corresponds to “an object area” in the presentinvention.

Image data, to which the result of detection of the face area carriedout by means of a photographing device such as a digital camera, etc.,has been added as metadata, is inputted to an input unit 101. Here, notethat the metadata of the face area to be added is the information whichcan specify the coordinates of vertices of a rectangular area whichindicates the existence of the face.

For example, they are the horizontal coordinate (x coordinate) and thevertical coordinate (y coordinate) of an upper left vertex of therectangular area indicating the existence of the face, and thehorizontal coordinate (x coordinate) and the vertical coordinate (ycoordinate) of a lower right vertex thereof. Here, note that in thisembodiment, description will be made by taking, as an example, the casewhere a face area is detected as a rectangular area including therein aface part inside an image, but the present invention can also be appliedto an image in which a face area is detected in a shape other than arectangular area.

A face area information extraction unit 102 extracts the coordinates offour vertices of the rectangular area in the form of the face area fromthe metadata of the image data inputted to the input unit 101. Inaddition, the center coordinates of the face area are calculated fromthe vertex coordinates of the face area thus extracted. The face areainformation extraction unit 102 outputs the extracted vertex coordinateinformation and the center coordinate information of the face area(information on the object area) to a face area coefficient decisionunit 106 and an image processing unit 107. The face area informationextraction unit 102 of this embodiment constitutes an “acquisition unit”of the present invention.

A division unit 103 spatially divides the image data inputted to theinput unit 101 into a plurality of divided areas. A divisional statisticacquisition unit 104 calculates an average brightness value asbrightness information for each of the divided areas. The averagebrightness value is calculated by adding the brightness values in eachdivided area in a cumulative manner, and by finally dividing the thuscumulatively added brightness values by the number of pixels in eachdivided area.

A local coefficient decision unit 105 decides a gamma curve, which isused at the time of carrying out tone correction of a pixel located atthe center of each of the divided areas, from the corresponding averagebrightness value calculated by the divisional statistic acquisition unit104. A tone correction coefficient for tone correction processing can becalculated based on the gamma curve thus obtained.

Here, note that the local coefficient decision unit 105 may decide agamma curve, which is used at the time of carrying out the tonecorrection of the pixel located at the center of each of the dividedareas, not based on the average or brightness value for each dividedarea, but based on a brightness histogram for each divided area. Forexample, a gamma curve capable of improving the tone characteristic(gradient) of an attention tone is determined by calculating a darkportion attention tone and a bright portion attention tone from thebrightness histogram. In addition, a gamma curve used at the time ofcarrying out the tone correction of the pixel located at the center ofeach of the divided areas may instead be decided based on colorinformation such as an R (red) histogram, a C (green) histogram, a B(blue) histogram, etc.

Here, description will be made to a decision method for gamma curvesused at the time of carrying out the tone correction of the individualcentral pixels of the divided areas by means of the local coefficientdecision unit 105, based on FIG. 2 and FIG. 3. FIG. 2 is a view in whichsix mutually adjacent divided areas 201 through 206 are extracted froman image divided by the division unit 103 as an example of divided areashaving common portions with the face area and those divided areas whichare located in the vicinity thereof.

In addition, the pixels located at the individual centers of the dividedareas are denoted by central pixels 207 through 212, respectively, andthe face area specified based on the coordinate information of the facearea which is inputted from the face area information extraction unit102 is denoted by a face area 213, and the pixel located at the centerof the face area is denoted by a central pixel 214.

Moreover, the pixels which exist in the vertex positions of the facearea 213 are denoted as face area vertex pixels 215 through 218,respectively, and an area of a predetermined range which touches theouter periphery of the face area 213 is denoted as a outer face area219, and the pixels which exist in the vertex positions of the outerface area 219 are denoted as outer face area vertex pixels 220 through223, respectively.

In addition, FIG. 3 shows gamma curves 301 through 305 which are storedin a gamma curve storage unit (not shown). Here, note that the gammacurve 301 with a steep slope in an area where the input value is smallis a gamma curve which is suitable at the time of assigning many tonesto a low brightness input. On the other hand, the gamma curve 305 with asteep slope in an area where the input value is large is a gamma curvewhich is suitable at the time of assigning many tones to a highbrightness input.

For example, the gamma curve 301 has a gamma coefficient k of 0.6; thegamma curve 302 has a gamma coefficient k of 0.8; the gamma curve 303has a gamma coefficient k of 1; the gamma curve 304 has a gammacoefficient k of 1.2; and the gamma curve 305 has a gamma coefficient kof 1.4.

Here, the local coefficient decision unit 105 decides gamma curves usedfor the tone correction of the central pixels 207 through 212 of thedivided areas by selecting them from the gamma curve storage unit (notshown) based on the average brightness values of the divided areas 201through 206 calculated by the divisional statistic acquisition unit 104.The following is mentioned as an example of such a selection method.

That is, four thresholds with which a size relation of Th1<Th2<Th3<Th4is satisfied are set as thresholds of the average brightness value, andin cases where the average brightness value is equal to or less than thethreshold Th1, the gamma curve 301 is selected. In cases where theaverage brightness value is larger than Th1 and equal to or less thanTh2, the gamma curve 302 is selected. In cases where the averagebrightness value is larger than Th2 and equal to or less than Th3, thegamma curve 303 is selected. In cases where the average brightness valueis larger than Th3 and equal to or less than Th4, the gamma curve 304 isselected. In cases where the average brightness value is larger thanTh4, the gamma curve 305 is selected.

In this embodiment, the local coefficient decision unit 105, whichdecides a gamma curve used in the central pixel of each of the dividedareas based on the average brightness value of each divided area,constitutes “a first decision unit” of the present invention.

The face area coefficient decision unit 106 decides a gamma curve, whichis used at the time of carrying out the tone correction of the centralpixel 214 of the face area and the outer face area vertex pixels 220through 223. This processing is carried out based on the gamma curves,which are used in the central pixels 207 through 212 of the dividedareas and which have been decided by the local coefficient decision unit105, and the vertex coordinate information and the center coordinateinformation of the face area, which are inputted from the face areainformation extraction unit 102.

First, specific reference will be made to a method of deciding the gammacurve used in the central pixel 214 of the face area. The central pixel214 of the face area exists in the rectangular area which has verticesat the central pixels 207, 208, 210, 211 of the divided areas.

Accordingly, the face area coefficient decision unit 106 determines agamma curve used for the tone correction of the central pixel 214 of theface area by carrying out the weighted interpolation of the gamma curvesused for tone correction in each of the central pixels 207, 208, 210,211 of the divided areas. This weighted interpolation is a weightedinterpolation according to the distance of the central pixel 214 of theface area to each of the central pixels 207, 208, 210, 211 of thedivided areas.

That is, the gamma curve used for the tone correction of the centralpixel 214 of the face area is decided based on the information of theface area, and the gamma curves of two divided areas which have a commonportion with the face area, and of two divided areas which are adjacentto these areas, among the divided areas. Specifically, such a decisionis made based on the gamma curves used for tone correction in thecentral pixels 208, 211 of the two divided areas, among the dividedareas, which have a common portion with the face area, and the gammacurves used for tone correction in the central pixels 207, 210 of thetwo divided areas contiguous to the above-mentioned two divided areas.

In this embodiment, the face area coefficient decision unit 106, whichdecides a gamma curve used for the tone correction of the central pixel214 of the face area, constitutes “a second decision unit” of thepresent invention.

Next, specific reference will be made to a method of deciding the gammacurves used in the outer face area vertex pixels 220 through 223. Theouter face area vertex pixels 220, 222 exist in the rectangular areawhich has vertices at the central pixels 207, 208, 210, 211 of thedivided areas.

Accordingly, the face area coefficient decision unit 106 determinesgamma curves used for the tone correction of the outer face area vertexpixels 220, 222 by carrying out the weighted interpolation of the gammacurves used for tone correction in each of the central pixels 207, 208,210, 211 of the divided areas. This weighted interpolation is a weightedinterpolation according to the distance of each of the outer face areavertex pixels 220, 222 to each of the central pixels 207, 208, 210, 211of the divided areas.

Similarly, the face area coefficient decision unit 106 determines gammacurves used for the tone correction of the outer face area vertex pixels221, 223 by carrying out the weighted interpolation of the gamma curvesused for tone correction in each of the central pixels 208, 209, 211,212 of the divided areas. This weighted interpolation is a weightedinterpolation according to the distance of each of the outer face areavertex pixels 221, 223 to each of the central pixels 208, 209, 211, 212of the divided areas.

According to the processing of the local coefficient decision unit 105and the face area coefficient decision unit 106 as referred to above,gamma curves, which are image processing coefficients (tone correctioncoefficients) used for the tone correction of the central pixels 207through 212 of the divided areas, the central pixel 214 of the facearea, and the outer face area vertex pixels 220 through 223, aredecided.

The image processing unit 107 carries out image processing (tonecorrection) on the image outputted from the input unit 101 based on theimage processing coefficients decided by the local coefficient decisionunit 105 and the face area coefficient decision unit 106, and the vertexcoordinates of the face area inputted from the face area informationextraction unit 102.

In the following, by using FIG. 2 and FIG. 4, reference will be made tothe image processing carried out by the image processing unit 107 in asequential manner while dividing it into three parts including the facearea 213, areas outside of the outer face area 219, and the outer facearea 219. FIG. 4 is a view showing a virtual gamma coefficient k′ ofpixels located on a line which connects between the pixel 208 and thepixel 211 of FIG. 2.

First, image processing in the face area 213 will be explained. Theimage processing unit 107 carries out image processing on all the pixelscontained in the face area 213 by the use of a common image processingcoefficient in order to make the image processing of the face area 213uniform.

Specifically, the image processing unit 107 carries out gamma conversionby uniformly applying the gamma curve to be used for the tone correctionof the central pixel 214 of the face area, which has been decided by theface area coefficient decision unit 106, to all the pixels contained inthe face area 213 without depending on the coordinates of the pixels. Asa result of this, the image processing of the face area, to whichviewers pay attention, becomes uniform, so it becomes possible to reducethe possibility of giving an odd or uncomfortable feeling to theviewers.

Then, the image processing in the areas outside of the outer face area219 will be explained. The image processing unit 107 carries out imageprocessing on individual pixels in the areas outside of the outer facearea 219 by the use of local gamma curves for obtaining the effect ofenhancing contrast or reducing local extinction of tone in dark regionand bright region.

Specifically, in the tone correction processing which is made on eachpixel which exists in the outside of the outer face area 219 and in therectangular area having its vertices at the central pixels 207, 208,210, 211, the image processing unit 107 performs the followingprocessing. That is, the image processing unit 107 first carries outgamma conversion on each pixel by the use of the gamma curves used forthe respective tone correction of the central pixels 207, 208, 210, 211of the individual divided areas.

Then, the image processing unit 107 performs the weighted sum averagingof the results of the above-mentioned gamma conversion carried out byuse of the gamma curves used for the respective tone correction of thecentral pixels of the individual divided areas to those pixels which areto be processed or subjected to tone correction processing, according tothe coordinates of the pixels to be processed. Specifically, the imageprocessing unit 107 carries out weighted sum averaging processing of theresults of the gamma conversion according to the distances from thepixels to be processed to each of the central pixels 207, 208, 210, 211of the individual divided areas, and outputs this (the weighted sumaverage thus obtained) as an image processing result (Lone correctionprocessing result) of the pixels to be processed.

Alternatively, the tone correction processing, which is made on eachpixel outside of the outer face area 219 and inside of the rectangulararea having its vertices at the central pixels 207, 208, 210, 211, maybe carried out as follows. First, the weighted sum averaging of thegamma coefficients used for the tone correction of the central pixels207, 208, 210, 211 is carried out according to the distances from thepixels to be processed in the tone correction processing to each of thecentral pixels 207, 208, 210, 211. This (the weighted sum average thusobtained) is used as a gamma coefficient k′ (this being called a virtualgamma coefficient) which is used for the tone correction of the pixelsto be processed.

Subsequently, gamma conversion is carried out on the pixels to beprocessed which exist inside of the rectangular area having its verticesat the central pixels 207, 208, 210, 211, by the use of the virtualgamma coefficient k′, and this (the result of the gamma conversion) isoutputted as an image processing result (a tone correction processingresult). The processing as mentioned above corresponds to “tonecorrection processing in which weighted interpolation is carried outaccording to coordinates, by the use of a tone correction coefficientfor each of divided areas” of the present invention.

Similarly, in the tone correction processing which is made on each pixelwhich exists in the outside of the outer face area 219 and inside of arectangular area having its vertices at the central pixels 208, 209,211, 212, the image processing unit 107 performs the followingprocessing. That is, the image processing unit 107 first carries outgamma conversion on each pixel by the use of the gamma curves used forthe tone correction of the individual central pixels 208, 209, 211, 212of the individual divided areas.

Then, the image processing unit 107 performs the weighted sum averagingof the results of the above-mentioned gamma conversion carried out byuse of the gamma curves used for the tone correction of the centralpixels of the individual divided areas to those pixels which are to beprocessed or subjected to tone correction processing, according to thecoordinates of the pixels to be processed. Specifically, the imageprocessing unit 107 carries out weighted sum averaging processing of theresults of the gamma conversion according to the distances from thepixels to be processed to each of the central pixels 208, 209, 211, 212of the individual divided areas, and outputs this (the weighted sumaverage thus obtained) as an image processing result (tone correctionprocessing result) of the pixels to be processed.

Alternatively, the weighted sum averaging of the gamma coefficients usedfor the tone correction of the central pixels 208, 209, 211, 212 iscarried out according to the distances from the pixels to be processedin the tone correction processing to each of the central pixels 208,209, 211, 212. This (the weighted sum average thus obtained) is used asa gamma coefficient k′ (a virtual gamma coefficient) which is used forthe tone correction of the pixels to be processed.

Subsequently, gamma conversion may be carried out on the pixels to beprocessed which exist inside of the rectangular area having its verticesat the central pixels 208, 209, 211, 212, by the use of the virtualgamma coefficient k′, and this (the result of the gamma conversion) maybe outputted as an image processing result (a tone correction processingresult).

In addition, with respect to the pixels in an area outside of arectangular area having its vertices at the central pixels 207, 209,210, 212, the image processing unit 107 carries out the same imageprocessing as mentioned above by using the central pixels (not shown) ofdivided areas which exist outside of the divided areas 201 through 206shown in FIG. 2.

That is, in the tone correction processing on individual pixels in theareas outside of the outer face area 219 as pixels to be processed, thegamma curves used for the tone correction of the central pixels of theindividual divided areas decided by the local coefficient decision unit105 are applied with weights being attached thereto according to thecoordinates of the pixels to be processed, respectively.

Thus, in this embodiment, the image processing unit 107 carries outweighted summation of the individual gamma curves used for the tonecorrection of the central pixels of the individual divided areas whichhave been decided by the local coefficient decision unit 105, whileweighting them according to the coordinates of the pixels to beprocessed. With this, a tone correction coefficient for each pixel inareas other than the face area will be decided as a result thereof.Accordingly, in this embodiment, the image processing unit 107, whichperforms such weighted sum processing, can also be said as constituting“the first decision unit” of the present invention.

As described above, in the areas outside of the outer face area 219, itbecomes possible to enhance contrast or to reduce local extinction oftone in dark region and/or local extinction of tone in bright region byperforming image processing with reference to the gamma curves decidedbased on the average brightness value or the brightness histogram of thedivided areas.

Here, the areas outside of the outer face area 219 constitute “an areaother than the object area” in the present invention. The outer facearea 219 also constitutes “an area other than the object area”, but inthis embodiment, still more detailed tone correction processing iscarried out on the outer face area 219, as will be described later.

Next, image processing in the outer face area 219 will be explained. Asmentioned above, in the face area 213, a tone correction coefficientapplied to each pixel in that area is decided with reference to a commongamma curve. In addition, in the areas outside of the outer face area219, a tone correction coefficient applied to each pixel in that area isdecided with reference to the gamma curves used in the tone correctionof the central pixels 207 through 212.

The outer face area 219 is an area for changing the tone correctioncoefficient related to image processing in a gradual manner whilepreventing the tone correction coefficient from changing in a rapidmanner, in a boundary between the face area 213 and the areas outside ofthe outer face area 219.

Specifically, in the tone correction processing which is made on thosepixels, as the pixels to be processed, which are included in aquadrilateral area having its vertices at the face area vertex pixels215, 216 and the outer face area vertex pixels 220, 221, the imageprocessing unit 107 performs the following processing. That is, theimage processing unit 107 first carries out gamma conversion on thepixels to be processed with reference to the gamma curves used for thetone correction of the pixels at the positions of the vertices 215, 216,220, 221.

Then, the image processing unit 107 performs the weighted sum averagingof the results of the above-mentioned gamma conversion carried out byapplying the gamma curves used for the tone correction of the individualvertices 215, 216, 220, 221 to those pixels which are to be processed orsubjected to tone correction processing, according to the distances ofthe pixels to be processed to the vertices 215, 216, 220, 221,respectively. This (weighted sum average thus obtained) is outputted asan image processing result (tone correction processing result) of thepixels to be processed.

Alternatively, the tone correction processing, which is made on thosepixels, as the pixels to be processed, which are included in thequadrilateral area having its vertices at the face area vertex pixels215, 216 and the outer face area vertex pixels 220, 221, may be carriedout in the following manner. First, the weighted sum averaging of thegamma coefficients used for the tone correction of the vertices 215,216, 220, 221 is carried out according to the distances from the pixelsto be processed in the tone correction processing to the vertices 215,216, 220, 221, respectively. This (the weighted sum average thusobtained) is used as a gamma coefficient k′ (a virtual gammacoefficient) which is used for the tone correction of the pixels to beprocessed.

Subsequently, gamma conversion is carried out on those pixels to beprocessed which are included in the quadrilateral area having itsvertices at the vertices 215, 216, 220, 221, by the use of the virtualgamma coefficient k′, and this (the result of the gamma conversion) isoutputted as an image processing result (a tone correction processingresult).

Here, note that the gamma curves used for the tone correction of theface area vertex pixels 215, 216 are equal to the gamma curve used forthe tone correction of the central pixel 214 of the face area, asmentioned above. In addition, the gamma curves used for the tonecorrection of the outer face area vertex pixels 220, 221 are those whichhave been decided by the face area coefficient decision unit 106.

The tone correction of the pixels in a quadrilateral area surrounded bythe face area vertex pixels 216, 218 and the outer face area vertexpixels 221, 223, and the tone correction of the pixels in aquadrilateral area surrounded by the face area vertex pixels 217, 218and the outer face area vertex pixels 222, 223 are carried out in asimilar manner. Also, the tone correction of the pixels in aquadrilateral area surrounded by the face area vertex pixels 215, 217and the outer face area vertex pixels 220, 222 is carried out in asimilar manner.

The tone correction in the outer face area 219 is carried out accordingto the above described processing with reference to both of the gammacurve used for the tone correction of the face area 213, and the gammacurve used for the tone correction of the areas outside of the outerface area 219.

That is, in the tone correction processing on individual pixels insideof the outer face area 219 as pixels to be processed, the gamma curvesdecided by the local coefficient decision unit 105 and the face areacoefficient decision unit 106 are applied with weights being attachedthereto according to the coordinates of the pixels to be processed,respectively. As a result, it becomes possible to achieve a gradualchange of the tone correction coefficient related to image processing inthe vicinity of the boundary between the face area 213 and the areasoutside of the outer face area 219.

As mentioned above, the image processing unit 107 carries out imageprocessing according to each of the face area 213, the areas outside ofthe outer face area 219, and the outer face area 219, and outputs imagedata obtained by the image processing to the output unit 108. The outputunit 108 outputs the image data, on which image processing has beencarried out by the image processing unit 107, to an external display.

FIG. 5 and FIG. 6 are flow charts showing image processing of thisembodiment. First, based on FIG. 5, reference will be made to a flow todecide a gamma curve in the form of an image processing coefficient. Aninput image is divided by the division unit 103 (S401), and a divisionalstatistic is acquired by the divisional statistic acquisition unit 104(S402), after which a gamma curve used for the tone correction of thecentral pixel of each of the divided areas is decided by the localcoefficient decision unit 105 (S403).

At this time, in cases where a face area exists in the image (S404), theface area coefficient decision unit 106 further decides gamma curves tobe used for the tone correction of a central pixel of the face area, andvertex pixels of an outer face area (S405).

Next, the flow of image processing will be explained by the use of FIG.6. In cases where a pixel to be processed or subjected to imageprocessing is one which exists in the face area (S501), the imageprocessing unit 107 performs gamma conversion processing on that pixelby the use of the gamma curve used for the tone correction of thecentral pixel of the face area (S502).

In cases where a pixel to be processed is one which exists in an areaoutside of the outer face area (S503), gamma conversion processing iscarried out on that pixel by the use of a gamma curve used for the tonecorrection of the central pixel of a corresponding divided area (S504).

In cases where a pixel to be processed or subjected to image processingis one which exists in the outer face area, gamma conversion processingis carried out on that pixel by the use of the gamma curve used for thetone correction of the central pixel of the face area and the gammacurves used for the tone correction of the vertex pixels of the outerface area (S505).

According to the above processing, it becomes possible to carry outuniform image processing on the face area while carrying out local imageprocessing with respect to the areas other than the face area. As aresult, it also becomes possible to prevent the viewers from being givenan odd or uncomfortable feeling due to nonuniform image processing onthe face area, without spoiling the effect of locally enhancing contrastor the effect of reducing extinction of tone in dark region and/orextinction of tone in bright region.

The image processing on pixels in the areas other than the face area(i.e., the outer face area and the area in the outside thereof) has beencarried out by performing the weighted sum averaging of the results ofgamma conversion carried out by the use of the gamma curves applied tothe tone correction of the central pixels of the divided areas where thepixels being processed exist, and by the use of the gamma curves appliedto the tone correction of the central pixels of their nearby dividedareas.

On the other hand, the image processing on pixels in the areas otherthan the face area (i.e., the outer face area and the area in theoutside thereof) may be carried out by performing gamma conversion onlyby the use of the gamma curves applied to the tone correction of thecentral pixels of the divided areas where the pixels being processedexist.

In this case, image processing on the pixels in the areas other than theface area is performed for each of the divided areas, so the processingis simplified. As mentioned above, an image processing coefficient isdecided for each of the divided areas based on the statistic of theimages in the divided areas, so even if simplified in this manner, localimage processing can be carried out in the areas other than the facearea.

Although an example in which one face area exists in image data has beendescribed in this embodiment, even in cases where there exist aplurality of face areas, the present invention can be applied byperforming similar processing with respect to each of the face area.

Further, as shown in FIG. 7, even in cases where there exist a pluralityof face areas and in cases where a plurality of face areas spatiallyoverlap one another, the present invention can be applied. For example,the present invention can be applied by newly calculating a joint facearea 603 (inclusion area) from overlapped face areas 601, 602, andcarrying out the above-mentioned image processing on the joint facearea.

At this time, the coordinates of an upper left vertex of the joint facearea 603 are a combination of the smallest horizontal coordinate (xcoordinate) and the smallest vertical coordinate (y coordinate) amonghorizontal coordinates (x coordinates) and vertical coordinates (ycoordinates) of four face area vertices of each of the face areas 601,602.

Also, the coordinates of a lower right vertex of the joint face area 603are a combination of the largest horizontal coordinate (x coordinate)and the largest vertical coordinate (y coordinate) among horizontalcoordinates (x coordinates) and vertical coordinates (y coordinates) ofthe four face area vertices of each of the face areas 601, 602.

It can be considered that information on such an inclusion area is notincluded in metadata. In that case, the face area information extractionunit 102 calculates inclusion area information based on each piece offace area information included in metadata. In addition, the imageprocessing unit 107 may perform tone correction on the pixels in theareas other than the inclusion area by carrying out sum averaging of thegamma conversion results, which have been obtained by the use of thegamma curves decided by the local coefficient decision unit 105, whileweighting them according to the coordinates of the pixels, similarly asdescribed above.

Second Embodiment

The first embodiment is an embodiment which, in the case of existence ofone or more face areas, makes the image processing coefficients whichare used in the individual face areas uniform respectively. This secondembodiment is an embodiment which makes the image processingcoefficients used for the tone correction of pixels in a face areauniform, and which, in the case of existence of a plurality of faceareas, makes image processing coefficients used for the tone correctionof pixels in each face area common in the plurality of face areas. Whenthis embodiment is applied, it becomes possible to carry out uniformimage processing over the plurality of face areas which exist in imagedata.

A block diagram for an image processing apparatus in this embodiment isthe same as that of FIG. 1. In addition, the contents of processing inthe input unit 101, in the face area information extraction unit 102, inthe division unit 103, in the divisional statistic acquisition unit 104,in the local coefficient decision unit 105, and in the output unit 108are the same as those in the first embodiment.

The face area coefficient decision unit 106 carries out the sameprocessing as in the first embodiment in cases where only one face areaexists, but in the case of existence of a plurality of face areas, theface area coefficient decision unit 106 carries out processing formaking the image processing coefficients used for the tone correction ofpixels in each face area common in the plurality of face areas. Thisprocessing will be explained based on FIG. 8 and FIG. 9.

FIG. 8 is a view showing, by way of example, a case where three faceareas 701 through 703 are inputted from the face area informationextraction unit 102 onto the six adjacent divided areas shown in FIG. 2.In addition, the pixels located at the centers of the three face areas701 through 703, respectively, are represented as central pixels 704through 706, respectively, and areas of a predetermined range, which arein touch with the outer peripheries of the individual face areas,respectively, are represented as outer face areas 707 through 709,respectively.

In addition, FIG. 9 shows gamma curves used for the tone correction ofthe three central pixels 704 through 706 as gamma curves 801 through803, respectively. The gamma curves used for the tone correction of theindividual central pixels are calculated by the method explained in thefirst embodiment.

First, the face area coefficient decision unit 106 calculates an averageof the gamma curves used for the tone correction of the pixels in theindividual face areas, in order to decide an image processingcoefficient to be used in common in the tone correction of the pixels inthe plurality of face areas. The gamma curves used for the tonecorrection of the pixels in the individual face areas 701 through 703are the gamma curves used for the tone correction of the central pixels704 through 706 of the individual face areas.

Accordingly, a gamma curve used in common in the tone correction of thepixels in the plurality of face areas is calculated by averaging thegamma curves 801 through 803 used for the tone correction of the centralpixels 704 through 706 shown in FIG. 8 by the number of the face areas.

Here, note that the average gamma curve thus calculated is shown as agamma curve 804 in FIG. 9. The face area coefficient decision unit 106takes the gamma curve 804 calculated in the above-mentioned manner as agamma curve which is used in common in the tone correction of the pixelsin the plurality of face areas.

Then, the image processing unit 107 carries out the same imageprocessing as in the first embodiment on individual areas including theface areas 701 through 703, areas outside of the outer face areas 707through 709, and the outer face areas 707 through 709 in a sequentialmanner.

According to the above processing, it becomes possible to make the imageprocessing coefficients used for the tone correction of the pixels inthe plurality of face areas existing in the image data common, so itbecomes possible to carry out uniform image processing over theplurality of face areas existing in the image data.

Here, note that in this embodiment, the local coefficient decision unit105 decides the gamma curve to be used from the average brightnessvalue, but a brightness histogram for each of the divided areas may beacquired by the divisional statistic acquisition unit 104, and the gammacurve may be decided according to information on the brightnesshistogram thus acquired.

In this embodiment, image processing should be performed by using thecommon image processing coefficient in all of the plurality of faceareas existing in the image data. That is, one group is defined by allof the plurality of face areas, and tone correction processing has beenperformed by using the common image processing coefficient in all theface areas belonging to the group.

In contrast to this, tone correction processing may be carried out byusing a common image processing coefficient only in a part of theplurality of face areas. That is, a group may be defined by a part ofthe plurality of face areas, and tone correction processing can beperformed by using the common image processing coefficient in all theface areas belonging to the group.

In this case, the face area coefficient decision unit 106 calculatesintervals between the plurality of face areas based on the centercoordinate information of the face areas inputted from the face areainformation extraction unit 102, and in cases where these intervals areeach larger than a predetermined threshold, a common use of the imageprocessing coefficient should not be made. The intervals between theface areas can be obtained, for example, by the distances between thecenter coordinates (representative points). That is, a group may bedefined as a set of face areas between which the distances are equal toor less than the predetermined threshold, among the plurality of faceareas.

Third Embodiment

The first embodiment and the second embodiment are ones in which localimage processing and uniform image processing on a face area(s) are madecompatible with each other with respect to image data to whichinformation on an area(s) in which a face(s) exists has been added asmetadata.

In this third embodiment, a specific object is detected by aphotographing device such as a digital camera, etc., and imageprocessing is carried out on image data to which information on the kindof the object and information on an area in which the object exists havebeen added as metadata. This embodiment achieves, in this imageprocessing, compatibility between local image processing and uniformimage processing on the object area.

A block diagram for an image processing apparatus in this embodiment isshown in FIG. 10. In FIG. 10, an input unit 101, a division unit 103, adivisional statistic acquisition unit 104, a local coefficient decisionunit 105, and an output unit 108 are the same as those in the firstembodiment. Here, note that as the specific object, there can be listedobjects such as, for example, the face of a pet such as a cat, a dog,etc., the entire body of a person, the entire body of a pet, a buildingthat has been registered beforehand, etc. In addition, the informationof the kind of an object to be described later is the information foridentifying the kind of an object as listed in the above examples.

By making reference to the metadata of the image data inputted to theinput unit 101, an object information extraction unit 901 first extractsthe information on the kind of an object added to the image data and theinformation on the position (coordinates) of the object in which theobject exists. Then, the object information extraction unit 901determines whether the object thus extracted is a target on whichuniform image processing is carried out.

This determination is made by recording beforehand a table, whichdescribes the kinds of objects to be used as a target on which uniformimage processing is carried out, in a storage unit (not shown), and bycomparing the information on the kind of the object thus extracted andthe table. As a result, in cases where the extracted object is a targeton which uniform image processing is carried out, the object informationextraction unit 901 outputs the coordinate information of the object toan object area coefficient decision unit 902.

The object area coefficient decision unit 902 performs the sameprocessing as that of the face area coefficient decision unit 106 in thefirst embodiment except for that the “face area” in the processing ofthe face area coefficient decision unit 106 is replaced with “an area inwhich the object exists” and which is inputted from the objectinformation extraction unit 901. An image processing unit 903 performsthe same processing as that of the image processing unit 107 in thefirst embodiment except for that the “face area” in the processing ofthe image processing unit 107 is replaced with “an area in which theobject exists” and which is inputted from the object informationextraction unit 901.

According to the above processing, the specific object is detected bythe photographing device such as a digital camera, etc., and it becomespossible to carry out suitable image processing on the image data towhich information on the kind of the object and information on the areain which the object exists have been added as metadata. That is, imageprocessing in which local image processing and uniform image processingto the object area are made compatible with each other becomes possible.

Here, note that in this embodiment, the object information extractionunit 901 determines, based only on the kind of the object as informationfor making a determination, whether uniform image processing is to becarried out, but the object information extraction unit 901 may performsuch a determination based on the size, position and the like of theobject as information for making a determination, in addition to thekind of the object.

In this case, a table which describes the relation among the kind, size,position and the like of the object used as a target on which uniformimage processing is carried out has been beforehand recorded in thestorage unit (not shown). Then, the object information extraction unit901 should make a determination as to whether an object is a target onwhich uniform image processing is carried out, by comparing theinformation on the kind of the object and the coordinate informationthereof with the table.

Here, note that the image processing apparatus of this embodiment isconfigured such that an image to which face area information has beenadded as metadata is inputted thereto, and the face area information isacquired from the metadata, but it may instead be configured such thatit acquires face area information by receiving the input of an imagehaving no metadata, and by carrying out face detection processing onthat image.

In addition, although in the above embodiments, description has beenmade by taking, as an example, the image processing apparatuses whichperform tone correction, the concept of the present invention can alsobe applied to, in addition to the image processing apparatusesperforming tone correction, an image processing apparatus which carriesout the image processing of changing the brightness and/or color of animage, for example. When such image processing is applied to a face areain a nonuniform manner, the brightness and color of a face will becomenonuniform, so viewers may have an odd or uncomfortable feeling, buteven in such a case, by applying the present invention, it is possibleto perform such image processing on the face area in a uniform manner.

Moreover, the image processing can be locally carried out on areas otherthan the face area, so it also becomes possible to change the brightnessand/or the color thereof in a local manner. In addition, although in theabove embodiments, reference has been made to the configuration in whichthe tone correction coefficient for each of the divided areas is decidedbased on the brightness information of that divided area, it is alsopossible to decide the tone correction coefficient based on imageinformation other than the brightness information, or based on thestatistic of an image, etc.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2010-001310, filed on Jan. 6, 2010, and Japanese Patent Application No.2010-221459, filed on Sep. 30, 2010, which are hereby incorporated byreference herein in their entirety.

What is claimed is:
 1. An image processing apparatus comprising aprocessor that is configured to: input image data from an externalapparatus; determine a plurality of images, said plurality of imagesmeeting respective predetermined conditions; determine a tone correctioncharacteristic for each of the plurality of images based on imagecharacteristic corresponding to each of the plurality of images; performtone correction processing to each of the plurality of images byuniformly applying the tone correction characteristic corresponding toeach of the plurality of images; and output image data, to which thetone correction processing is performed in each of the plurality ofimages, to a display unit, wherein the processor is configured todetermine different tone correction characteristic for each of theplurality of images, in a case where the image characteristiccorresponding to each of the plurality of images are different eachother.
 2. The image processing apparatus of claim 1, wherein the tonecorrection characteristic is a gamma coefficient.
 3. The imageprocessing apparatus of claim 1, wherein each of the plurality of imagesis an image in which a predetermined kind of object exists.
 4. The imageprocessing apparatus of claim 1, wherein the image characteristic isbrightness.
 5. The image processing apparatus of claim 1, the processoris configured further to: divide an input image based on the input imagedata into a plurality of divided images; and decide a tone correctioncharacteristic for each of the divided images based on imagecharacteristic of each of the plurality of divided images, wherein theprocessor is configured to determine the tone correction characteristicfor each of the plurality of images based on the tone correctioncharacteristic of at least divided images which have a common portionwith each of the plurality of images.
 6. An image processing apparatuscomprising a processor that is configured to: input image data from anexternal apparatus; determine a plurality of images based on the imagedata; uniformly perform image correction processing to each of theplurality of images based on image characteristic corresponding to eachof the plurality of images; and output image data, to which the imagecorrection processing is performed in each of the plurality of images,to a display unit, wherein the processor is configured to performdifferent image correction processing to each of the plurality ofimages, in a case where the image characteristic corresponding to eachof the plurality of images are different each other.
 7. The imageprocessing apparatus of claim 6, wherein the image correction processingis a gamma correction processing.
 8. The image processing apparatus ofclaim 6, wherein each of the plurality of images is an image in which apredetermined kind of object exists.
 9. The image processing apparatusof claim 6, wherein the image characteristic is brightness.
 10. Acontrol method for an image processing apparatus comprising: inputtingimage data from an external apparatus; determining a plurality ofimages, said plurality of images meeting respective predeterminedconditions; determining a tone correction characteristic for each of theplurality of images based on image characteristic corresponding to eachof the plurality of images; performing tone correction processing toeach of the plurality of images by uniformly applying the tonecorrection characteristic corresponding to each of the plurality ofimages; and outputting image data, to which the tone correctionprocessing is performed in each of the plurality of images, to a displayunit, wherein different tone correction characteristic is determined foreach of the plurality of images, in a case where the imagecharacteristic corresponding to each of the plurality of images aredifferent each other.
 11. The control method for an image processingapparatus of claim 10, wherein the tone correction characteristic is agamma coefficient.
 12. The control method for an image processingapparatus of claim 10, wherein each of the plurality of images is animage in which a predetermined kind of object exists.
 13. The controlmethod for an image processing apparatus of claim 10, wherein the imagecharacteristic is brightness.
 14. The control method for an imageprocessing apparatus of claim 10, further comprising: dividing an inputimage based on the input image data into a plurality of divided images;and deciding a tone correction characteristic for each of the dividedimages based on image characteristic of each of the plurality of dividedimages, wherein in the determining the tone correction characteristic,the tone correction characteristic for each of the plurality of imagesis determined based on the tone correction characteristic of at leastdivided images which have a common portion with each of the plurality ofimages.
 15. A control method for an image processing apparatuscomprising: inputting image data from an external apparatus; determininga plurality of images based on the image data; uniformly performingimage correction processing to each of the plurality of images based onimage characteristic corresponding to each of the plurality of images;and outputting image data, to which the image correction processing isperformed in each of the plurality of images, to a display unit, whereindifferent image correction processing is performed to each of theplurality of images, in a case where the image characteristiccorresponding to each of the plurality of images are different eachother.
 16. The control method for an image processing apparatus of claim15, wherein the image correction processing is a gamma correctionprocessing.
 17. The control method for an image processing apparatus ofclaim 15, wherein each of the plurality of images is an image in which apredetermined kind of object exists.
 18. The control method for an imageprocessing apparatus of claim 15, wherein the image characteristic isbrightness.